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Abstract 

New  warfighting  concepts  are  currently  under  development  to  improve  the  ability  of  future  Joint  Force 
Commanders  to  rapidly  and  decisively  conduct  particularly  challenging  and  important  operational 
missions  as  they  transition  to  the  fighting  force  described  in  Joint  Vision  2020.  This  paper  describes  one 
element  that  is  part  of  these  new  concepts:  knowledge  management  and  collaboration  as  conducted  to 
support  effects-based  operations.  Collaboration  offers  great  potential  to  better  enable  warfighters  to 
plan,  monitor,  execute,  and  assess  activities  across  the  spectrum  of  joint  functional  areas.  Collaboration 
is  also  essential  to  develop  a  shared  situational  awareness  among  heterogeneous,  distributed  team 
members.  Effects-based  operations  is  defined  as  a  “process  for  obtaining  a  desired  strategic  outcome,  or 
‘effect’  on  the  enemy,  through  the  synergistic  and  cumulative  application  of  the  full  range  of  national 
(military  and  nonmilitary)  capabilities  at  all  levels  of  conflict."  This  paper  reports  on  the  results  of  a 
survey  administered  during  an  experiment  conducted  to  help  refine  the  effects-based  planning  process. 
Suggestions  for  improving  knowledge  management  include  developing  business  rules  for  working  in  a 
collaborative  information  environment  and  improved  display  capabilities  to  help  planners  track 
information  and  documents  during  different  phases  of  the  planning  process. 

1.  Introduction 

New  warfighting  concepts  are  currently  under  development  at  U.S.  Joint  Forces  Command  (JFCOM), 
J9,  Joint  Experimentation  Directorate,  to  be  employed  by  the  U.S.  military  as  it  transitions  to  the 
fighting  force  described  in  Joint  Vision  2020.  Their  primary  focus  is  to  develop  new  joint  warfighting 
concepts  and  capabilities  that  will  improve  the  ability  of  future  joint  force  commanders  to  rapidly  and 
decisively  conduct  particularly  challenging  and  important  operational  missions.'  A  series  of  experi¬ 
ments  and  exercises  is  being  conducted  to  help  refine  the  new  concepts  and  processes  under  develop¬ 
ment  at  JECOM.  Each  experiment  is  designed  to  support  the  assessment  of  these  proposed  future 
capabilities  and  modify  current  doctrine,  organization,  training,  materiel,  leadership,  personnel,  and 
facilities.  This  paper  describes  one  aspect  that  is  part  of  these  new  concepts:  knowledge  management 
and  collaboration  as  conducted  to  support  the  effects-based  planning  and  assessment  (EBP&A)  process. 
Collaboration  is  also  essential  to  develop  a  common  situational  awareness  among  heterogeneous, 
distributed  team  members. 


The  research  reported  here  was  sponsored  by  the  U.S.  Joint  Forces  Command,  J9,  Joint  Experimentation  Center,  Suffolk, 
VA. 


2.  Collaboration 


Collaboration  offers  great  potential  to  better  enable  warfighters  to  plan,  monitor,  execute,  and  assess 
activities  across  the  spectrum  of  joint  functional  areas.  Capabilities  included  in  collaboration  tools 
include  the  ability  to  share  applications,  have  a  virtual  workspace,  use  voice/audio,  whiteboard,  video, 
and  chat  functions.  Many,  if  not  all,  of  the  benefits  to  be  accrued  from  participating  in  a  face-to-face 
meeting  can  be  gained  using  collaborative  tools:  information  flows  quickly,  outstanding  issues  are 
raised,  and  a  certain  amount  of  brainstorming  can  occur  to  arrive  at  a  decision.  Additionally,  "all 
relevant  users  or  providers  of  information  reach  a  fuller  understanding  of  the  issues  because  they  have 
seen  other  viewpoints  and  received  a  freer  flow  of  information."^’  ^  ^  Collaborative  tools  offer  the  added 
capabilities  of  providing  the  ability  to  share  information  and  resources  and  coordinate  among  individuals 
across  geographic  and  temporal  boundaries. 

Specific  functions  needed  in  collaborative  tools  include  the  ability  to  accommodate  a  wide  variety  of 
group  interactions  including  one-to-one,  one-to-many,  many-to-many,  and  many-to-one.*''*  In  addition 
they  need  to  offer  the  flexibility  to  support  formal,  informal,  and  ad  hoc  collaborations.  Moreover,  these 
tools  should  be  intuitive  to  the  user  regarding  where  to  find  collaborators  and  how  to  interact  with  them. 
Collaborative  tools  should  also  accommodate  real  life  situations  such  as  interruptions  and  still  ensure 
work  can  be  resumed  seamlessly. 

The  following  taxonomy  of  collaborative  tools  was  included  in  the  Joint  Staff  report  on  collaborative 
tools 

•  same  time/same  place  (i.e.,  group  decision  support  and  meeting  facilitation) 

•  same  time/different  place  (i.e.,  audio/video  conferencing,  shared  whiteboards,  chat,  video/audio 
broadcast  and  shared  applications) 

•  different  time/same  place  (i.e.,  team  rooms,  shared  memory,  information  sharing,  coordination 
tools,  e-mail,  newsgroups  and  recorded  video/  audio) 

•  different  time/different  place  (i.e.,  e-mail,  newsgroups,  discussion/bulletin  boards,  group 
authoring,  recorded  video/audio  and  workflow) 

Table  1  presents  a  categorization  of  collaborative  tools  as  they  are  used  relative  to  time  and  place. 

Table  1.  Collaborative  Tool  Use  Relative  to  Time  and  Place.  * 


TIME 

SAME 

DIFFERENT  but 
PREDICTABLE 

DIFFERENT  and 
UNPREDICATBLE 

PLACE 

SAME 

Meeting 

Facilitation 

Work  Shifts 

Team  Rooms 

DIFFERENT  but 
PREDICTABLE 

Tele/Video/ 

Desktop 

Conferencing 

E-mail 

Collaborative 

Writing 

DIFFERENT  and 
UNPREDICATBLE 

Interactive 

Multicast 

Seminars 

Computer 

Bulletin 

Boards 

Workflow 

^  From:  Joint  Collaboration  Tools  Employment,  Joint  Staff,  Washington,  D.C.  CJCSM  bxxx.Ol,  2001. 


2.1  Definitions 


The  following  definitions  of  the  functionality  afforded  by  collaborative  tools  are  from  the  Joint  Staff 
report  on  joint  collaborative  tools  employment: 

Application  sharing:  gives  an  organization  the  capability  to  use  one  copy  of  an  application  for 
multiple  use,  which  saves  space  and  ensures  application  uniformity. 

Chat  Room:  Allows  members  of  a  workgroup  to  "text  talk"  among  themselves,  see  what  each  is 
saying  and  have  a  record  of  the  conversations. 

Document  Management:  A  tool  that  facilitates  acquiring,  classifying,  storing,  and  accessing 
organizational  documents. 

Document  Sharing:  Allows  a  workgroup  to  share  the  same  document  from  one  location,  giving 
everyone  the  most  up-to-date  version. 

E-mail:  E-mail  allows  the  user  to  pick  and  choose  the  recipients  of  the  package  being  sent.  The 
drawback  of  e-mail  is  that  any  information  sent  is  sent  individually  to  each  recipient  and  can  use 
large  amounts  of  bandwidth  to  carry  out  each  transaction. 

Events  Eog:  The  events  log  provides  an  area  where  a  log  for  a  workgroup  is  kept  as  an  official  re¬ 
cord  of  the  chronology  and  participants  in  key  events  that  take  place  during  a  collaborative  event. 

Intelligent  Agent:  An  intelligent  agent,  or  simply  agent,  is  a  program  that  gathers  information  or 
performs  some  other  service  without  a  user's  immediate  presence  on  a  regular  schedule.  It  utilizes 
parameters  the  user  has  set  up,  searches  and  gathers  information  that  is  useful  (determined  by  the 
parameters  of  the  search  criteria),  and  presents  it  to  the  user  at  a  specific  time. 

Metadata:  Metadata,  or  information  about  data,  provides  users  with  descriptions  of  the  data  or 
informational  objects  they  can  access.  This  metadata  is  stored  with  the  document  and  is  used  to 
accelerate  searches  for  data. 

Push  Technology  (Netcasting):  Software  that  sends  notification  of  changes  to  controlled  documents, 
keeping  all  participants  informed  of  the  latest  version. 

Scheduler/Calendar:  Gives  a  workgroup  a  central  location  to  post  meeting  times,  project  milestones 
and  other  information  that  is  time-related.  Allows  users  to  schedule  meetings  by  seeing  free  and 
busy  times  for  other  participants. 

Virtual  Domain:  Gives  an  organization  the  flexibility  to  set  up  workgroups  that  are  not  collocated. 

2.2  Benefits  of  Computer-Assisted  Collaboration  for  New  Military  Concepts 

Additional  benefits  afforded  by  collaborative  tools  that  are  especially  germane  to  the  new  military 
concepts  currently  under  development  include:  allowing  smaller  deployed  warfighter  footprints; 
improving  information  exchange  among  decisionmakers  and  sources  of  information;  including 
distributed  subject  matter  experts  as  participants  in  the  decisionmaking  process;  capitalizing  on  the 
synergy  of  the  total  command  and  control  (C2)  infrastructure  during  operations  versus  relying  on 
isolated  efforts;  and  improving  decisionmaking  timelines. 

3.  Effects-Based  Operations 

Effects-Based  Operations  (EBO)  is  defined  as  a  “process  for  obtaining  a  desired  strategic  outcome,  or 
‘effect’  on  the  enemy,  through  the  synergistic  and  cumulative  application  of  the  full  range  of  national 
(military  and  nonmilitary)  capabilities  at  all  levels  of  conflict.”  •’  An  effect  is  the  physical, 
functional,  or  psychological  outcome,  event,  or  consequence  that  results  from  specific  military  or  non¬ 
military  actions.  The  EBO  concept  is  based  on  the  tenant  that  a  better  understanding  of  the  adversary. 


and  the  increased  involvement  of  other  national  agencies,  will  lead  to  better-reasoned  options  to  engage 
potential  adversaries. 

The  EBO  process  is  conceptualized  as  a  continuous  and  iterative  planning  and  execution  cycle.  This 
cycle  comprises  the  following  five  phases.  Developing  a  comprehensive  knowledge,  or  insight,  into  the 
nature  of  the  adversary,  the  environment,  and  our  own  capabilities;  this  is  accomplished  by  conducting 
an  operational  net  assessment  (this  process  is  described  below).  The  second  phase  entails  articulating  the 
desired  effects  necessary  to  break  the  adversary’s  cohesion  and  to  cause  the  adversary  to  change  his 
behavior — these  effects  are  articulated  via  an  Effects  Tasking  Order  (ETO).  The  third  phase  involves 
determining  and  applying  those  elements  of  national  power,  i.e.,  diplomatic,  informational,  military,  and 
economic,  that  will  be  most  effective  in  achieving  the  desired  effects,  and  assessing  the  application  of 
capabilities  in  terms  of  the  desired  effects.  The  forth  phase  includes  conducting  an  integrated  and 
continuous  assessment  process  to  measure  and  assess  the  impact  of  the  effects  created.  The  fifth  phase 
involves  making  decisions  regarding  ways  the  commander  can  adapt  and  adjust  the  current  course  of 
action  to  more  effectively  reach  the  desired  end  state. 

Characteristics  that  distinguish  effects-based  planning  from  traditional  objectives-based  planning 
include  development  of  a  broader  and  deeper  insight  into  the  adversary  through  the  fusion  of 
information  from  a  broad  spectrum  of  sources,  including  national  and  international,  government  and 
non-govemmental.  Insight  into  the  adversary  is  developed  through  conducting  a  complex  “system  of 
systems  analysis”  (SoSA)  of  the  adversary.  Planners  conduct  this  systems  analysis  collaboratively  with  a 
networked  and  distributed  team  of  experts  (cultural,  behavioral,  technical,  economic,  political,  and 
military)  and  centers  of  excellence  (think  tanks  and  other  institutions  where  expertise  resides  relevant  to 
the  adversary)."^  EBO  planning  goes  a  step  further  than  consideration  of  which  actions  will  achieve  an 
objective  (i.e.,  the  traditional  approach)  to  consider  the  full  range  of  potential  results  of  our  actions 
(including  direct  and  indirect  effects,  desired  and  undesired  effects).  Key  aspects  of  EBO  are  the  ability 
of  decisionmakers  to  quickly  recognize  any  unexpected  effects  and  the  flexibility  and  agility  to  adapt  to 
the  implications  of  those  effects.  A  second  key  aspect  of  EBO  is  an  emphasis  on  the  ability  to  examine 
the  causal  linkages  and  effects  through  which  actions  lead  to  objectives.  A  third  aspect  is  the  Joint  Task 
Eorce  (JTE)  Commander's  broadened  focus.  This  ability  to  more  precisely  select  the  right  set  of  actions 
is  expected  to  offer  the  Joint  Eorce  Commander  the  ability  to  further  minimize  undesired  collateral 
effects. 

Joint  activities  involved  in  EBO  would  be  inefficient  without  the  use  of  collaborative  tools.  Planners 
and  decisionmakers  in  distributed  locations  need  to  conduct  synchronous  dialog  sessions  with  each  other 
and  interact  based  on  the  use  of  a  shared  view  of  the  battlespace  concerning  the  impact  of  actions 
included  in  the  ONA  and  ETO.  A  common  understanding  of  the  impact  of  proposed  actions,  the  desired 
effects  to  be  achieved,  responsibilities  for  carrying  out  the  actions,  and  the  current  friendly  and 
adversary  situation  are  also  accomplished  via  the  coordination  and  negotiation  activities  that  are 
conducted  in  a  collaborative  information  environment. 

4.  Operational  Net  Assessment 

The  challenge  inherent  in  the  EBO  process  is  to  accurately  identify  the  causal  linkages  that  determine 
whether  or  not  the  action  taken  will  achieve  the  desired  effect.*'’''^  This  challenge  is  addressed  through 
the  Operational  Net  Assessment  (ONA).  The  ONA  is  a  product  of  collaboration  among  analysts  at 
strategic,  operational,  and  tactical  levels.  It  provides  a  common  knowledge  base  available  to  all 
customers  from  the  national  strategic  level  to  the  tactical  level.  Contributions  of  potential  customers 


impact  each  element  of  the  ONA,  since  the  ONA  views  a  potential  adversary  as  an  interdependent 
system  of  systems,  all  of  which  contribute  to  some  degree  toward  his  societal  coherence,  will,  and 
capability  to  pursue  a  course  of  action  inimical  to  friendly  interests.  The  ONA  goes  far  beyond 
traditional  intelligence  assessments.  It  is  an  action-oriented  process  that  provides  a  continuous  stream  of 
knowledge  from  adversary  vulnerabilities  to  effects  to  tasks.  Information  contained  in  the  ONA  should 
help  planners  and  operators  develop  greater  situational  awareness  and  understanding  of  the  adversary. 

5.  Limited  Objective  Experiment  on  Effects  Tasking  Order-to-Actions 

An  experiment  entitled  Effects  Tasking  Order  (ETO)-to-Actions  Eimited  Objective  Experiment  (ETO- 
to-Actions  EOE),  was  conducted  at  the  U.S.  Joint  Eorces  Command  (JECOM),  Joint  Eutures  Eabora- 
tory,  Suffolk,  VA,  3-14  December,  2001,  to  examine  elements  of  EBO,  and  to  specifically  assess  and 
refine  the  effects-based  planning  and  assessment  (EBP&A)  processes.  This  experiment  was  designed 
and  conducted  in  a  partnership  with  the  Naval  Postgraduate  School,  JECOM,  J9,  and  the  Navy  Warfare 
Development  Command.  Previous  experiments  and  exercises,  such  as  Unified  Vision  01,  focused  pri¬ 
marily  on  the  Joint  Task  Eorce  Headquarters  (JTEHQ)  level  of  command,  with  the  majority  of  effort 
directed  at  refining  the  process  down  to  producing  an  ETO.  The  ETO-to- Actions  EOE  focused  within 
and  below  the  JTEHQ  to  examine  the  required  coordination  and  collaboration  processes  (both  vertically 
and  horizontally)  between  the  JTEHQ  and  functional  components  headquarters,  which  is  needed  to 
collaboratively  develop  the  ETO  and  translate  the  effects  directed  in  it  into  tactical  actions  on  the 
battlefield. 

The  experimental  participants  included  representatives  from  all  services,  organized  into  a  Joint  Task 
Eorce  HQ  staff  and  functional  component  HQ  staffs  (Joint  Eorce  (JE)  Maritime  Component  Commander 
(JEMCC),  JE  Air  Component  Commander  (JEACC),  JE  Eand  Component  Commander  (JEECC),  Joint 
Special  Operations  Task  Eorce  (JSOTE),  and  Joint  Psychological  Operations  Task  Eorce  (JPOTE)  (not 
depicted  in  Eigure  1)).  Thirty-nine  percent  of  the  participants  were  active  duty,  forty-two  percent  were 
retired  military  personnel,  and  the  remainder  were  civilian  concept  developers  at  JECOM.  Three  days  of 
training  on  the  new  effects-based  planning  processes  and  the  collaboration  and  decision  support  tools 
were  provided.  Eive  days  of  game  execution  consisted  of  the  participants  conducting  collaborative 
planning  between  the  Joint  Task  Eorce  HQ  staff  and  the  subordinate  functional  component  staffs,  with 
the  JTE  Commander  issuing  new  and  revised,  collaboratively  developed,  ETOs  to  the  Joint  Eorce. 
Eigure  1  depicts  the  organizational  structure  that  used  was  for  the  EOE  and  the  number  of  participants  in 


Figure  1.  Organizational  structure  and  number  of  participants  for  tbe  ETO-A  LOE.  ^ 


5.1  Effects-Based  Planning  Process 


Figure  2  depicts  a  high-level  view  of  the  steps  entailed  in  the  effects-based  planning  (EBP)  process  and 
the  planning  steps  that  were  the  focus  of  this  LOE.  [Note,  the  steps  in  the  EBP  process  are  indicated  by 
the  rectangles;  the  planners,  who  perform  these  steps,  are  indicated  by  the  circles.]  The  experiment 
began  when  the  Commander-in-Chiefs  (CINC)  mission  statement  was  issued.  The  first  step  in  the  EBP 
process  involved  completing  a  mission  analysis,  as  a  collaborative  (vertical  and  horizontal)  process 
between  the  Joint  Planning  Cell  (comprised  of  representatives  from  the  JTEHQ,  the  components,  the 
CINC  staff,  and  the  Inter-agency  Coordination  Section)  and  the  component’s  planning  cells. 
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Figure  2.  Elements  of  the  JTF  Effects-Based  Planning  process  included  in  the 
Effects  Tasking  Order-to-Actions  Limited  Objective  Experiment.® 

5.2  Information  Technology  and  Collaboration  Tools  Used  in  the  ETO-to-Actions  LOE 

The  ability  to  engage  in  real-time  collaboration  with  other  decisionmakers  who  are  geographically 
distributed,  and  having  rapid  access  to  current,  relevant,  and  accurate  information,  have  become 
indispensable  elements  of  the  joint  command  and  control  (C2)  planning  and  decisionmaking  process.  A 
collaboration  tool  suite  was  introduced  to  facilitate  these  information-intensive  interactions  during  the 
ETO-to-Actions  EOE.  New  information  technology  (IT)  tools,  used  as  part  of  a  networked,  web-based 
collaborative  system  were  also  introduced.  One  goal  for  the  experiment  was  to  develop  an 
understanding  of  the  implications  and  effects  of  the  distributed  planning  that  provides  the  foundation  for 
an  EBO. 

A  web-based  tool,  called  the  Wargaming  Information  Grid  System  (WIGS),  and  a  collaborative 
planning  environment,  the  Information  Workspace  (IWS),  provided  the  core  for  the  collaboration  tool 


suite.  IWS  was  used  within  and  among  the  functional  and  service  components.  Functions  provided  by 
IWS  include  real-time  text  chat  and  voice,  both  conducted  over  the  internet.  All  occupants  within  a 
virtual  “room”  (within  the  IWS  tool)  could  view  and  respond  interactively  to  an  initiator’s  message. 
WIGS  was  designed  to  be  the  central  source  for  shared  information  during  scenario  play.  The  objectives 
for  this  website  were  to:  (1)  provide  a  location  to  post  information  for  all  players  and  to  facilitate  the 
exchange  of  documents,  (2)  provide  access  to  analytic  tools  and  to  an  underlying  database  of  reference 
and  briefing  materials,  as  well  as  (3)  provide  links  to  additional  websites,  in  order  to  provide  additional 
information  related  to  game  play. 

The  following  tools  were  provided  to  participants,  in  addition  to  WIGS  and  IWS:  ADOCS,  ONA  Tool, 
and  JDIM.  The  automated  deep  operations  coordination  system  (ADOCS)  is  a  joint  service  mission 
management  software  application  that  presented  a  visual  display  of  the  battlespace.  ADOCS  provides  a 
suite  of  tools  and  interfaces  for  horizontal  and  vertical  integration  across  battlespace  functional  areas. 
The  operational  net  assessment  (ONA)  tool  was  accessed  through  WIGS  and  provided  an  initial 
assessment  of  the  military  operation  from  multiple  points  of  view.  These  other  perspectives  on  the 
operation  included  information  and  analysis  from  other  branches  of  the  U.S.  Government,  and  many 
other  sources  of  information,  as  well  as  the  requirements  for  information  from  non-govemmental 
organizations.  The  Joint  Distributed  ISR  Management  Tool  (JDIM)  enabled  the  user  to  find  the 
intelligence,  surveillance,  and  reconnaissance  (ISR)  information  that  existed  at  particular  nodes  in  the 
organization. 

5.3  Survey  Administration 

Three  surveys  were  administered  during  the  experiment  to  gather  data  on  the  effectiveness  of  (1)  the 
collaborative  tools  and  the  training  provided  to  the  participants  for  this  experiment,  (2)  knowledge 
management  and  collaboration  as  critical  aspects  of  effects-based  planning,  and  (3)  the  effects-based 
planning  and  assessment  process.  This  paper  describes  the  results  of  the  Knowledge  Management  and 
Collaboration  Survey,  and  is  a  companion  paper  to  two  other  papers  that  discuss  results  of  the  other  two 

6,  7 

surveys. 

Twenty-nine  survey  items  were  presented  as  statements  about  managing  information  and  collaborating 
to  conduct  the  effects-based  planning  process.  (Due  to  space  limitations,  a  sub-set  of  these  items  are 
discussed  in  this  paper  and  are  listed  in  Table  1.)  Participants  were  asked  to  rate  the  extent  to  which 
they  agreed  with  each  statement  by  using  a  five-point  Lickert  scale  that  ranged  from  “strongly  disagree” 
to  “strongly  agree”  (where  1  =  strongly  disagree,  2  =  somewhat  disagree,  3  =  neither  disagree  nor  agree, 
4  =  somewhat  agree,  and  5  =  strongly  agree).  Five  of  the  items  were  open-ended  questions  and  asked 
participants  to  indicate  ways  in  which  various  aspects  of  the  process  could  be  improved.  The  survey 
was  administered  to  all  99  experimental  participants;  90  completed  surveys  were  returned  and  analyzed. 

6.  Results 

Table  2  presents  the  survey  items,  the  mean  ratings  given  by  the  participants,  and  the  standard  deviation 
for  each  rating.  Overall,  the  participants'  ratings  for  the  majority  of  the  survey  items  were  above  the 
midpoint  on  the  five-point  rating  scale  that  was  used  for  these  items.  This  is  interpreted  to  mean  the 
participants  found  the  event  to  be  a  productive  learning  experience  and  the  experiment  contributed  to 
their  developing  an  understanding  of  the  effects-based  planning  and  assessment  (EBP&A)  process.  Each 
of  the  items  will  be  discussed  in  the  order  in  which  they  are  listed  in  the  table. 


Table  2.  Knowledge  Management  and  Collaboration  Survey  Questions  and  Results. 


Item  Survey  Question 

Number 

Mean 

Rating 

Std 

Dev. 

1 

My  cell/center  was  able  to  maintain  good  situational  awareness. 

3.8 

1.1 

2 

I  was  able  to  maintain  good  situational  awareness  across  the  entire  JTF/  Component 
organization. 

3.3 

1.1 

3 

The  new  JTF  structure  allowed  Components  to  obtain  adequate  and  timely  information. 

3.3 

.96 

4 

The  Effects-Based  Planning  Process  allowed  components  to  obtain  adequate  and  timely 
information. 

3.2 

.99 

5 

What  important  pieces  of  information  did  you  need  but  where  unable  to  find? 

_  * 

_  * 

6 

When  unable  to  find  information  you  needed,  what  were  the  causes? 

_  A 

_  A 

7 

How  were  the  interactions  required  across  the  JTF  to  conduct  EBP&A  different  from 
what  would  have  traditionally  be  done? 

_  * 

_  * 

8 

Participation  in  asynchronous  collaboration  sessions,  while  engaging  in  other  tasks,  was 
an  effective  way  to  conduct  planning. 

3.0 

1.2 

9 

The  new  JTE  organization  structure  facilitated  the  EBP&A  process. 

3.5 

.9 

10 

The  JTE  Effects  Assessment  Cell  is  a  necessary  element  of  the  EBP&A  process. 

3.7 

1.1 

11 

The  JTF  System  of  Systems  Analysis  (SoSA)  Cell  is  a  necessary  element  of  the  EBP&A 
process. 

3.7 

1.1 

12 

The  JTE  Collection  Management  Cell  is  a  necessary  element  of  the  EBP&A  process. 

4.0 

.94 

13 

Collaboration  improved  my  situational  awareness. 

4.2 

1.0 

14 

The  ONA  enabled  me  to  develop  a  better  understanding  of  the  adversary  as  a  result  of  the 
fusion  of  information  from  a  spectrum  of  sources. 

3.4 

1.1 

15 

The  ONA  facilitates  component-level  planning. 

3.5 

.86 

16 

Our  cell  was  able  to  conduct  the  parallel  collaborative  planning  that  was  closely 
coordinated  with  subordinate  commands. 

3.5 

.98 

17 

The  presentation  aspects  of  the  information  available  provided  an  enhanced 
understanding  of  the  situation,  which  facilitated  rapid  development  of  the  ETO. 

Yes=59% 

No=41% 

Notes.  *  Indicates  a  text  only  response. 


^  Indicates  a  different  set  of  choices  were  presented  for  this  item:  these  are  discussed  in  section  5.6. 

6.1  Maintaining  Situational  Awareness 

The  question  that  asked  about  their  ability  to  maintain  good  situational  awareness  (SA)  within  their  cell 
received  a  mean  rating  of  3.8,  indicating  that  the  majority  felt  they  could  maintain  good  SA.  One  player 
stated  that  the  tools,  e.g.,  ADOCS,  provided  very  good  SA,  however,  the  level  of  play  for  the  experi¬ 
ment  was  not  that  demanding.  In  contrast,  one  player  thought  the  time  constraints  in  the  LOE  impacted 
their  ability  to  maintain  good  SA  and  having  only  one  screen  to  manage  information  was  a  limitation. 

6.2  Maintaining  situational  awareness  across  the  entire  JTF /Component  organization 

The  mean  rating  for  the  question  that  asked  about  their  ability  to  maintain  good  SA  across  the  entire  JTF 
organization  was  3.3,  slightly  lower  than  their  ability  to  maintain  SA  within  their  cell.  This  is  not 
surprising,  as  maintaining  SA  on  the  entire  JTF  organization  is  considerably  more  demanding.  There 
was  a  wide  range  in  their  ratings  depending  on  where  they  were  located  within  the  organization. 

6.3  New  JTF  structure  allowed  components  to  obtain  adequate  and  timely  information 


The  statement  that  the  new  JTF  structure  allowed  components  to  obtain  adequate  and  timely  information 
received  a  mean  rating  of  3.3,  indicating  they  neither  agreed  nor  disagreed  with  this  statement.  Those 
who  gave  this  item  a  low  rating  indicated  it  was  difficult  to  know  who  was  responsible  for  what  action, 
which  piece  of  information  posted  on  WIGS  was  authoritative,  and  the  limitations  of  the  experiment 
(e.g.,  the  concepts  are  still  under  development)  precluded  timely  information  flow.  Some  players 
indicated  they  did  not  think  the  structure  was  very  new  or  that  the  structure  provided  the  information; 
instead,  it  was  the  tools  that  were  new  and  provided  the  information.  Comments  pertaining  to  the 
inability  to  obtain  adequate  and  timely  information  indicated  the  lack  of  analysis  [capability]  in  the 
automated  tools,  the  inability  to  navigate  the  tools  (due  to  insufficient  familiarity  with  the  tools),  and  not 
knowing  where  to  mine  the  information  were  problems. 

6.4  EBP&A  process  allows  components  to  obtain  adequate  and  timely  information 

The  assertion  that  the  EBP&A  process  allowed  components  to  obtain  adequate  and  timely  information 
received  a  mean  rating  of  3.2,  indicating  that  participants  did  not  particularly  agree  or  disagree  with  this 
statement.  Several  players  made  the  distinction  between  effects-based  planning,  which  is  much  more 
developed,  conceptually,  and  effects-based  assessment,  which  needs  further  development. 

6.5  Important  information  participants  were  unable  to  find 

Many  things  were  listed  in  response  to  the  item  that  asked  what  important  pieces  of  information  they 
needed  but  were  unable  to  find.  The  most  important  information  was  the  overall  system  of  systems 
perspective  and  the  relationship  between  systems  and  nodes  in  time  and  space.  Other  information  listed 
as  needed  included:  a  workflow  status  indication  for  where  they  were  in  the  overall  planning  process, 
knowledge  about  the  actions  of  other  components  that  were  occurring  concurrently  and  that  was 
required  prior  to  another  component  executing  a  specified  mission,  highlighting  the  enemy's  capabilities 
(the  ONA  focuses  on  the  enemy's  vulnerabilities),  weaknesses  in  friendly  courses  of  action,  effects 
achieved  from  current  operations,  a  complete  nodal  analysis  with  weighting  of  nodes  and  effects,  and 
timing  considerations  to  maximize  the  effect  of  candidate  actions.  This  information  was  seen  as 
essential  to  support  the  overall  objective  of  developing  a  stronger  plan  by  using  these  new  effects-based 
processes.  A  suggestion  for  a  knowledge  management  capability  was  to  provide  links  between  the 
listing  of  forces/equipment/systems  in  theatre  and  the  capabilities  of  those  units. 

6.6  Reasons  for  inability  to  obtain  needed  information 

Problems  that  caused  the  inability  to  obtain  all  the  needed  information  included  the  following  categories: 
not  knowing  where  to  get  the  information  (21%),  not  enough  time  to  obtain  the  information  (29%), 
information  needed  was  not  available  from  any  source  (22%),  inadequate  responses  to  information 
queries  (3%),  inadequate  communications  (4%)  and  other  reasons  (18%).  Several  players  indicated  the 
cause  was  due  to  databases  not  being  fully  populated.  As  is  typical  with  an  event  of  this  size,  there  was  a 
wide  range  of  opinions  expressed.  Some  players  felt  that  there  was  not  enough  time  to  think,  and  locate 
information,  while  others  thought  the  pace  of  events  was  fairly  slow.  Several  players  thought  the 
information  was  not  displayed  in  a  way  that  facilitated  its  use  or  linked  the  information  to  related 
information  in  a  way  that  facilitated  its  use  in  a  timely  manner.  A  comment,  that  goes  beyond  the 
limitations  of  this  experiment,  was  the  issue  of  the  quality  of  information,  i.e.,  given  the  tremendous 
amount  of  information  available,  or  potentially  available,  how  will  information  be  vetted? 


6. 7  How  interactions  to  conduct  EBP&A  were  different  from  what  is  traditionally  done 


The  item  that  asked  how  the  interactions  required  across  the  JTF  to  conduct  EBP&A  were  different  from 
the  way  planning  is  traditionally  done  elicited  a  wide  range  of  replies.  The  information  being  looked  at 
is  different,  since  the  focus  is  on  'effects'  and  nodes  vice  "checking  off  targets  that  were  hit."  Focusing 
on  the  desired  effect  also  required  a  better  understanding,  by  the  components,  as  to  exactly  what  the  JTF 
Commander  and  staff  had  in  mind.  Many  players  added  that  the  main  difference  is  the  emphasis  on  a 
collaborative  planning  environment  that  allows  parallel  planning  versus  serial  planning.  Use  of  the 
ONA,  the  greater  emphasis  devoted  to  developing  measures  of  effectiveness  (MOEs)/measures  of 
performance  (MOPs),  including  the  standing  joint  command  and  control  element,  and  the  JFMCC 
organization,  were  all  different  from  traditional  operations. 

6.8  Effectiveness  of  engaging  in  asynchronous  collaboration  sessions  to  conduct  planning 

The  effectiveness  of  engaging  in  asynchronous  collaboration  sessions,  while  engaging  in  other  tasks,  as 
an  effective  way  to  conduct  planning,  received  a  mean  rating  of  3.0.  Asynchronous  planning  refers  to 
collaboration  that  occurs  at  different  times,  when  participants  in  the  collaboration  process  are  involved  in 
other  tasks  in  addition  to  the  task  on  which  they  are  collaborating.  One  party  places  the  information  they 
desire  to  collaborate  on  in  a  particular  location,  then  the  second  party,  at  a  later  time,  digests  the  material, 
and  responds.  Shared  data  is  the  key  to  collaboration,  where  two  or  more  people  with  a  common 
problem  use  shared  data  to  produce  a  product.  This  equivalency,  regarding  the  effectiveness  of  engaging 
in  asynchronous  collaboration  sessions  to  conduct  planning,  is  attributed  to  several  factors:  the 
participants'  lack  of  familiarity  with  conducting  planning  this  way,  their  lack  of  familiarity  with  the  other 
people  in  the  collaboration  sessions,  the  newness  of  the  tools  and  processes,  and  that  they  were 
concurrently  engaged  in  other  tasks.  Several  participants  thought  engaging  in  asynchronous  collabora¬ 
tion  sessions  was  too  distracting  while  trying  to  concentrate  on  the  task  at  hand;  others  became  "task- 
saturated"  as  they  participated  in  several  concurrent  chat  sessions.  Engaging  in  a  discussion  in  one 
collaborative  session  while  monitoring  the  other  on-going  collaborative  sessions,  and  concurrently 
fielding  questions  from  a  player  in  another  chat  session  spread  their  attention  across  too  many  tasks. 
Others  thought  this  the  way  planning  will  be  done  in  the  future  in  a  collaborative  information 
environment,  and  that  this  is  a  skill  that  needs  to  be  learned  like  any  other  skill.  In  a  real  setting,  many  of 
the  participants  would  know  each  other,  and  would  have  a  history  of  collaborating.  These  "habitual 
relationships"  are  recognized  as  critical  to  being  able  to  engage  in  effective  collaboration,  i.e.,  when 
people  collaborating  know  each  other,  there  are  fewer  problems  of  trust  and  understanding.  Some 
players  wanted  to  reserve  judgment  on  this  question  given  the  artificialities  of  the  experimental 
environment. 

6.9  Extent  to  which  new  JTF  organization  facilitated  the  EBP&A  process 

The  item  that  asked  whether  the  new  JTF  organization  structure  facilitated  the  EBP&A  process  received 
a  mean  rating  of  3.7,  indicating  that  it  did  for  the  majority  of  the  experimental  audience.  While  the 
EBP&A  process  is  maturing,  the  operations  part  and  the  assessment  process  were  not  given  enough 
play. 

6.10  JTF  Effects  Assessment  Cell  is  a  necessary  element  of  the  EBP&A  process 

The  assertion  that  the  JTF  effects-assessment  cell  is  a  necessary  element  of  the  EBP&A  process  received 
a  mean  rating  of  3.9,  indicating  people  agreed  with  this  statement.  Participants  who  agreed  stated: 
Feedback  is  absolutely  necessary,  assessment  is  the  key  to  EBO,  continuous  assessment,  and  dynamic 
re-tasking  is  an  integral  part  of  the  operation.  Assessment  drives  the  application  of  resources  to 


accomplish  the  desired  effect.  One  participant  thought  that  a  pure  intelligence  analyst  may  not  have  the 
operational  experience  to  do  this  job,  and  certainly  not  in  an  intelligence  cell.  Someone  with  an 
intelligence  background  would  be  valuable  because  this  job  requires  creating  and  selecting  indicators, 
and  intelligence  people  already  do  that  as  part  of  their  job. 

6.11  JTF  System  of  Systems  Analysis  (SoSA)  Cell  is  a  necessary  element  of  EBP&A 

The  item  that  asked  whether  the  JTF  system  of  systems  analysis  (SoSA)  cell  is  a  necessary  element  of 
the  EBP&A  process  received  a  mean  rating  of  3.7,  indicating  people  agreed  with  this  statement.  One 
player  stated  that  the  SoSA  cell  provides  the  fundamental  background  on  how  to  go  about  achieving  the 
effects;  however,  he  cautioned  it  is  not  an  intelligence  cell  and  not  a  targeting  cell.  Several  players 
agreed,  but  had  concerns  about  where  this  function  would  be  located,  e.g.,  it  should  be  a  dedicated 
capability  (possibly  federated),  removed  from  the  JTF,  with  routine  access  to  all  available  information 
and  expertise. 

6.12  The  JTF  collection  management  cell  is  a  necessary  element  of  the  EBP&A  process 

The  item  that  asked  if  the  JTF  collection  management  cell  is  a  necessary  element  of  the  EBP&A  process 
received  a  mean  rating  of  4.0,  indicating  people  agreed  with  this  statement.  A  player  who  thought  the 
role  was  conceptually  appropriate,  thought  it  focused  too  much  on  the  usual  airborne  and  overhead 
imaging  assets,  particularly  those  that  are  historically  low  density/high  demand.  He  thought  this 
emphasis  was  tied  to  the  old  way  of  doing  things  and  "does  not  seem  to  effectively  integrate  with  other 
warfighting  missions,  tasks,  and  effects." 

6.13  Collaboration  improves  situational  awareness 

The  assertion  that  collaboration  improved  situational  awareness  received  a  mean  rating  of  4.2,  indicating 
the  majority  agreed  with  this  statement.  Many  players  made  positive  comments  in  response  to  this  item. 
A  succinct  statement  was  that  "information  being  passed  during  collaboration  always  improves  your 
SA."  Several  felt  that  collaboration  is  a  key  element  of  developing  SA  and  that  collaboration  is  the  [way 
of  the]  “future.”  Players  who  did  not  agree  with  this  statement  thought  collaboration  was  done  in  a  very 
narrow  band  or  that  the  time  they  spent  in  collaborative  sessions  was  not  very  rewarding.  Comments 
related  to  the  tools  were  that  most  SA  came  from  collaboration  (as  opposed  to  the  tools)  indicting 
improved  tools  are  needed. 

6.14  ON  A  contributes  to  developing  a  better  understanding  of  the  adversary 

The  assertion  that  the  ONA  contributed  to  developing  a  better  understanding  of  the  adversary  received  a 
mean  rating  of  3.4,  indicating  people  did  not  particularly  agree  with  this  statement.  The  most  important 
types  of  information  that  were  missing  were  the  overall  system  of  systems  (SoSA)  perspective  and  the 
relationship  between  systems  and  nodes,  in  time  and  space.  Several  players  indicated  the  ONA,  as  used 
in  the  experiment,  did  not  provide  a  better  understanding  because  there  was  no  single  place  to  go  for 
various  types  of  information  and  they  had  to  search  for  information.  One  player  indicated  the  process  of 
building  the  intelligence  preparation  of  the  battlefield,  by  all  staff  sections,  created  better  SA  than 
hunting  and  individually  looking  for  intelligence  "nuggets."  The  inherent  complexity  of  the  ONA  was 
another  issue.  Some  players  didn’t  think  the  underlying  information  and  links  were  developed 
sufficiently  to  provide  a  thorough  understanding  of  the  adversary.  The  ONA  has  the  potential  of 
contributing  to  planners  developing  a  better  understanding  of  the  adversary,  but  (as  the  concept  of 
operations  indicates)  users  of  this  information  require  knowledge  [of  the  information  included  in  the 


ONA]  prior  to  the  start  of  the  scenario  and  they  did  not  have  that  benefit  during  the  LOE.  The  ONA 
needs  more  cross  effects,  cross  actions,  and  cross  node  linkages  [added  to  it]... as  it  stands  now,  the 
ONA  is  just  a  database. 

6.15  ONA  facilitates  Component  level  planning 

The  item  that  asked  if  the  ONA  facilitates  component-level  planning  received  a  mean  rating  of  3.5.  One 
player  thought,  “we  have  enacted  semi-automatic  mechanical  linkages  from  the  ONA  down  to  tactical 
level  planning.  What  is  missing  is  any  SoSA  input  to  the  links  that  are  in  the  ONA.”  In  general, 
comments  indicated  the  ONA  could  facilitate  component-level  planning  if  there  were  better  linkages  and 
collaboration  across  components.  Several  players  thought  the  ONA  could  facilitate  component  level 
planning  if  the  ONA  was  supported  by  analysis.  Those  who  gave  a  high  rating  to  this  item  felt  the  ONA 
offered  many  benefits,  including:  it  reduces  a  huge  volume  of  information  and  provides  a  critical  focus 
for  the  components;  it  provides  planning  guidance  for  the  components  to  use  in  translating  strategy  to 
operations;  tied  with  a  good  joint  intelligence  preparation  of  the  battlefield  (JIPB)  this  would  be  a  great 
tool;  the  ONA  will  facilitate  component  level  planning  (if  we  can  get  real  information  during  a  real 
world  crisis). 

6.16  Ability  to  conduct  parallel  planning  closely  coordinated  with  subordinate  commands 

The  item  that  asked  if  their  cell  was  able  to  conduct  the  parallel  collaborative  planning  that  was  closely 
coordinated  with  subordinate  commands  received  a  rating  of  3.5.  Several  players  felt  the  collaboration 
tools  supported  component  planning.  However,  another  player  felt  the  collaboration  between  the 
headquarters  and  the  components  was  superficial,  i.e.,  there  was  little  detail. 

6.17  Presentation  of  the  information  provided  an  enhanced  understanding  of  situation 

This  item  asked  if  the  presentation  aspects  of  the  information  available  provided  an  enhanced  under¬ 
standing  of  the  situation,  to  facilitate  rapid  development  of  the  ETO.  The  majority  of  the  players  (59%) 
indicated  "yes"  and  the  remaining  players  (41%)  indicated  "no."  Again,  several  players  indicated  the 
JIPB  was  essential  for  developing  a  campaign  plan.  One  player  indicated  the  ETO  web  page  was  useful, 
although  there  was  no  way  to  ascertain  its  currency.  ADOCS  was  listed  as  a  great  graphical  presen¬ 
tation.  Player  suggestions  included:  integrating  ADOCS  with  the  ONA  database,  adding  the  ability  to 
tailor  the  presentation  in  ADOCS,  and  including  presentation  techniques  to  show  the  relationship  of 
effects  and  key  nodes.  The  goal  for  this  integrated  presentation  would  be  to  be  able  to  select  and  effect  a 
node,  then  graphically  see  all  the  nodes  in  ADOCS  that  were  affected,  and  vice  versa.  Several  players 
indicated  they  need  the  ability  to  see  other  component  level  information  in  addition  to  each  cell’s  and 
component's  view  of  the  desired  effects  because  this  information  helps  others  adjust  to  support  the 
desired  effect. 

7.  Discussion 

Collaboration  offers  great  potential  to  better  enable  warfighters  to  plan,  monitor,  execute,  and  assess 
activities  across  joint  functional  areas  during  effects-based  operations.  Analysts  at  strategic,  operational, 
and  tactical  levels  need  to  collaborate  to  produce  the  recurring  and  non-recurring  products  that  support 
EBO.  Both  the  EBO  process  and  producing  the  planning  products  rely  on  extensive  collaboration 
between  planners  and  decisionmakers  at  all  levels — strategic  to  tactical.  Moreover,  the  more  extensive 
analysis  entailed  in  the  new  EBP&A  processes,  i.e.,  where  a  much  wider  range  of  information  is 
analyzed,  creates  a  more  challenging  information  management  task  for  planners  and  operators.  This 


challenge  can  be  met  with  the  capabilities  provided  by  a  collaborative  tool  environment.  The 
collaboration  that  occurred  between  the  players  during  their  participation  in  multiple  planning  cells 
facilitated  their  developing  and  maintaining  a  high-level  of  SA.  However,  many  players  expressed 
concerns  about  the  level  of  workload  encountered.  This  high  workload  was  produced  as  a  result  of 
several  factors.  These  factors  included  the  players  lack  of  famili-arity  with  the  new  EBP&A  processes, 
not  enough  time  allowed  to  accomplish  all  the  tasks,  and  the  amount  of  time  spent  "pulling"  data, 
indicating  a  greater  need  for  information  to  be  pushed  to  the  user  of  the  information.  The  complexity  of 
the  experimental  environment  precluded  determining  if  the  level  and  depth  of  collaboration  envisioned 
for  future  military  operations  will  produce  excessive  workload  levels. 

The  ability  to  collaborate  with  all  involved,  for  example,  to  adjust  the  track  of  a  critical  ISR  platform 
was  viewed  as  a  tremendous  capability.  Participation  in  asynchronous  collaboration  sessions,  while 
engaging  in  other  tasks,  was  considered  to  be  an  effective  way  to  conduct  planning  by  most  players. 
The  planning  and  deconfliction  required  the  exchange  of  a  great  deal  of  data,  which  was  greatly 
enhanced  by  working  within  a  collaborative  information  environment. 

8.  Conclusions 

The  experiment  participants  provided  many  valuable  suggestions  that  will  contribute  to  refining  the 
EBP&A  process.  Concerns  were  raised  about  the  new  procedures  creating  increased  workload.  The 
expectation  is  that  as  planners  and  operators  become  familiar  with  the  new  processes  and  tools  the 
workload  will  become  more  manageable.  Many  players  appreciated  the  value  added  by  the  new 
EBP&A  processes  in  terms  of  the  more  comprehensive  nature  of  the  analysis  of  the  adversary  and  the 
increased  speed  of  planning  that  is  possible  when  planning  is  done  collaboratively,  however,  the 
complexity  involved  caused  some  concern  for  some  players.  A  brief  summary  of  several  themes  that 
emerged,  that  cut  across  many  of  the  survey  items  is  discussed  below.  These  themes  included  the  need 
for  (1)  business  rules  for  planning  and  operating  within  a  collaborative  information  environment,  (2) 
improved  analysis  tools  and  displays,  and  (3)  additional  training  on  using  the  new  information 
technology  and  collaboration  tools. 

8.1  Business  rules  need  to  be  developed 

Business  rules  are  needed  to  facilitate  users  gaining  quick  access  to  critical  data  and  supporting 
documentation.  Eor  example,  information  was  posted  in  the  IWS  file  cabinet,  on  WIGS,  in  the  ONA 
tool,  on  SharePoint,  on  the  'T  drive,  and  on  individual  computers.  Eocating  essential  information 
became  difficult  when  there  was  no  established  procedure  for  information  storage.  Moreover, 
documents  located  within  any  one  of  these  systems  were  not  labeled  in  a  way  to  make  it  easy  to  know 
which  version  was  the  most  current,  or  what  the  content  was.  Business  rules  also  need  to  be  developed 
to  provide  standards  for  collaborating  among  the  participants,  for  example,  to  add  discipline  to  the 
process.  Clarification  is  needed  regarding  who  uses  which  tools,  and  when,  in  the  planning  (and/or 
execution)  process,  to  produce  which  products,  during  the  EBP&A  process.  Business  rules  are  needed 
regarding  when  and  where  to  post  information  and  to  increase  efficiency  by  reducing  the  need  for 
decisionmakers  to  search  through  tools  and  data  to  extract  the  needed  information. 

8.2  Provide  a  Synthesized  Picture 

A  major  theme  expressed  by  the  participants  was  the  need  for  synthesized  presentations  of  information, 
both  within  the  various  tools  used  and  across  the  tools.  Many  players  felt  the  inability  to  see  a  holistic 


picture  of  the  adversary,  and  how  striking  certain  nodes  would  affect  the  adversary,  was  an  impediment. 
Players  want  dynamic  tools  that  reflect  the  status  of  effects  as  they  are  achieved.  This  could  be  in  the 
form  of  a  visual  graphic  on  a  map  with  a  drill  down  hyperlink. 

8.3  Need  Additional  Training 

Players  indicated  they  were  not  sufficiently  familiar  with  the  new  processes  to  sufficiently  realize  the 
inherent  capabilities  of  the  new  EBP&A  process.  There  were  also  instances  where  players  did  not  know 
where  to  go  to  get  information  from  the  “system.”  More  explicit  training  on  the  new  processes  prior  to 
starting  the  experiment  would  facilitate  a  better  assessment  of  the  process. 
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